Perhaps better DX is to have directories and just cargo run --release in each?

@Firestar99 FYI, I am going to revamp this a bit as it is a bit of a mess.

FWIW, I am almost ready to update this. It is much cleaner now :-)

I think this is ready for review. Sorry it is so big! I can break it down if need be.

The main changes:

1. Move compiletests under tests/ and add difftests there as well.
2. Add a difftest harness
3. Add a difftest sdk/library used by the binaries under test.
4. Add a simple compute difftest test using wgpu (one variant written in rust, one in wgsl).
5. Add to CI (on linux install software GPU so the tests will run)
6. Some docs in a README.

This is what a run with passing and failing tests looks like:

On failing tests, the outputs are grouped as you may have 4 variants where 3 are the same and 1 is different. In that case the 3 same will be grouped in the output so it is easier to understand what is going on.

Another option we could do rather than binaries, is sort of what I did before (have built in harnesses which are configured) but mirror wgpu's harness and config format. Perhaps that would allow us to leverage their test corpus by just writing the rust-gpu versions of their test inputs.

If I get my naga rust-gpu backend working, we could generate the rust-gpu code on the fly from the source wgpu test 🤔

This doubles CI time. I am sure there are things I can do to make CI faster (such as the deps_helper in compile test) but perhaps we should do that in followups?

I pulled out the tests/* to tests/compiletests/* rename as a separate commit and rebased it for you.

In your previous rebase you forgot to move some of the new tests. I would like to fast-track that rename in #270 so it doesn't happen again.

I really like the concept and I love that we can diff rust-gpu, wgsl and rust on cpu or whatever else we can think of.

I've played around with it a bit and noticed a few things:

• I assume all difftests are in a different workspace to not pollute our root workspace and slow down cargo due to evaluating the potentially hundreds of cargo projects in difftests? Would love to see a sentence about it in the readme, there it just states it's done this way but not why it is.
• One crate is called difftest and the other difftests, confusing!
• compile times
  • FIXED: The slowest part of our project is compiling the c++ of spirv_tools, which we are building twice? Once to build difftest itself, run it and it building the tests builds it again? I assume we'd only need to compile it once, as we're only compiling rust shaders once as well.
  • Due to it being a different workspace, we can't share the spirv_tools or wgpu build with the main workspace. I tried setting the target dir to the same dir, didn't help, though the next issue may be affecting my results.
  • FIXED: Repeated runs still take 60s to run due to it recompiling something from scratch every time, as I see many rustc spinning up. This needs to be fixed if we ever want to iterate on bugs using difftests.
• ci
  • ci is not checking whether the new Cargo.lock is up-to-date with cargo fetch --locked
  • Maybe we should split out difftests into a separate runner to speed up builds? Maybe do the same with compiletests? That I'd make a followup to allow for more experimentation.
  • FIXED: the difftests are not using feature use-installed-tools due to being build by difftests and not directly from ci

(continually adding to this)

• I assume all difftests are in a different workspace to not pollute our root workspace and slow down cargo due to evaluating the potentially hundreds of cargo projects in difftests? Would love to see a sentence about it in the readme, there it just states it's done this way but not why it is.

Yep, added text to that effect.

• One crate is called difftest and the other difftests, confusing!

Yeah, I tried to match compiletest (they are called compiletests for the runner and https://crates.io/crates/compiletest_rs introduces a compiletest crate used by the runnner). Happy to change naming to make it less confusing.

• Maybe we should split out difftests into a separate runner to speed up builds? Maybe do the same with compiletests? That I'd make a followup to allow for more experimentation.

Yes, I think we should. I was planning to do so in a followup. I wanted to keep this close to "as-is" so others can see where I just mirrored the existing comptiletest support.

I'm working on the repeated runs, and noticed that there's some target_dir stomping going on. This is a clean run, and still the spirv-builder cargo run is marking crates that are supposedly clean as dirty:

     Running `tests/target/release/simple-compute-rust /tmp/.tmphUE36K`
   Compiling compiler_builtins v0.1.138
   Compiling core v0.0.0 (/home/firestar99/.rustup/toolchains/nightly-2024-11-22-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core)
       Dirty proc-macro2 v1.0.94: the rustflags changed
   Compiling proc-macro2 v1.0.94
   Compiling libm v0.2.11
       Dirty unicode-ident v1.0.18: the rustflags changed
   Compiling unicode-ident v1.0.18
       Dirty autocfg v1.4.0: the rustflags changed
   Compiling autocfg v1.4.0
   Compiling spirv-std-types v0.9.0 (/home/firestar99/workspace/frameworks/rust-gpu/crates/spirv-std/shared)

run3_clean.log

We could provide pre-made binary crates that one can just drop files in for simple cases. I'd still want to keep the top-level "run cargo run --release" API for ultimate flexibility, but maybe we move some straightforward "stock" runners into the runner binary itself rather than the way I did it where they are a library used by the test binaries and everything is cargo run --release. Not sure which way is better 🤷

The stomping is happening cause spirv-builder can only find it's target_dir of target/spirv-builder when it has the "OUT_DIR" env var, which is only present if a build script is present. Otherwise, it defaults to not setting it and stomping with the cpu compile.
We have an overwrite for the target_dir directory, but it only works if spirv-builder found the target dir and then it makes it relative to that. If you were to pass in an absolute path, it would just use that, ignoring the resolved value, but still requiring that it could be resolved.
I fixed it by making spirv-builder "accept absolute target_dir_path and properly resolve relative ones with cargo_metadata", which does introduce another dependency, though we have cargo_metadata in cargo-gpu already.
From 60s of stomping we're now down to about 2s of cpu compile and 2s shader compile incrementally.

I have one concern about the design of this system: If we look at the main functions of simple_compute_rust and *_wgsl, they are effectively the same, apart from the shader they use. With the current system, we'd need to ensure that the wgsl and rust versions dispatch the shader with the same hypercube, it could as well be the same main binary with a flag to select which shader to use. While copy-pasting those few lines may not be a huge deal, if we start to copy-paste a whole setup for a graphics pipeline, that's an issue.
Alternatively, we could have one crate depend on the other, but then the one crate's main would just be calling a single function by supplying it a different shader. Seems silly to have a crate just for that.
Instead, I'd propose to have a single crate per "test case" containing both rust and wgsl shaders. The amount of variants would have to be discovered by running the binary without any args, and with args it'll run that variant (just an enum?). The individual runs are still isolated from each other by rerunning the binary each time, while simplifying code sharing a lot. And if you wanted to add a CPU variant to the current system, you could just branch off the cpu variant before you get to the gpu variants.
You could even argue that every test could just be a binary within a single global crate.

I assumed any code being shared by more than 2 tests would be put into the difftest crate for reuse. The reason I did it this was was for maximum control (let's say you wanted different deps or different rustflags, of even different entries in cargo.toml). But it could definitely be collapsed into one. I'm indifferent, I just wanted to make sure we could test MxN scenarios